Method and apparatus with pose prediction

ABSTRACT

A processor-implemented pose prediction method includes: estimating a pose of a user for a predetermined period of time in real time; calculating an estimation confidence of the estimated pose; determining a weight for the pose of the user based on the estimated pose and the estimation confidence; and predicting a pose of the user after the predetermined period of time, based on the weight.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. § 119(a) of KoreanPatent Application No. 10-2021-0010136 filed on Jan. 25, 2021, in theKorean Intellectual Property Office, the entire disclosure of which isincorporated herein by reference for all purposes.

BACKGROUND 1. Field

The following description relates to a method and apparatus with poseprediction.

2. Description of Related Art

In various augmented reality (AR) devices, latencies may occur due to,for example, physical limitations caused by sensing, rendering, anddisplay processes. Due to a motion-to-photon latency occurring in suchan AR environment, a dynamic matching error between a virtual object anda real object according to an actual movement of a user may occur. Forexample, when a simultaneous localization and mapping (SLAM) scheme isused to estimate a pose of a user, an estimation error compared toground-truth information may be included in an input value. Also, sincea head motion of a user is nonlinear, the time required to estimate amotion may increase. In addition, in the case of a large movement of auser, overshooting may occur, which may cause a great matching errorduring matching between a virtual object and a real object.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In one general aspect, a processor-implemented pose prediction methodincludes: estimating a pose of a user for a predetermined period of timein real time; calculating an estimation confidence of the estimatedpose; determining a weight for the pose of the user based on theestimated pose and the estimation confidence; and predicting a pose ofthe user after the predetermined period of time, based on the weight.

The method may include: calculating a prediction error between theestimated pose and the predicted pose, based on the estimationconfidence; and tuning an artificial neural network (ANN) model byfeeding back the prediction error, the ANN model being configured topredict a pose of the user.

The calculating of the prediction error may include calculating theprediction error based on a comparison result between the estimationconfidence and a predetermined threshold.

The calculating of the prediction error based on the comparison resultmay include: calculating the prediction error in response to theestimation confidence being greater than the threshold; and adjustingthe prediction error based on a comparison result between the estimatedpose and a pose that is estimated for a time adjacent to thepredetermined period of time for which the pose is estimated, inresponse to the estimation confidence being less than or equal to thethreshold.

The tuning of the ANN model may include tuning a parameter of the ANNmodel by numerically calculating a gradient of the prediction error.

The tuning of the ANN model may include tuning a parameter of the ANNmodel using a least square scheme by grouping poses with relatively highestimation confidences as a reference set.

The calculating of the estimation confidence may include calculating anuncertainty of the estimated pose, based on an estimation covarianceaccording to a simultaneous localization and mapping (SLAM) scheme witha Kalman filter.

The calculating of the estimation confidence may include calculating anuncertainty of the estimated pose, based on a residual value between theestimated pose and a value obtained by approximating the estimated poseusing a curve fitting function.

The predicting of the pose may include: refining the estimated pose byadjusting the weight; and predicting the pose of the user after thepredetermined period of time based on the refined pose, the estimationconfidence, and the prediction error that is fed back.

The determining of the weight may include: setting the weight to begreater than a reference value in response to the estimation confidencebeing greater than a predetermined threshold; and adjusting the weightbased on a pose that is estimated for a time adjacent to thepredetermined period of time for which the pose is estimated, inresponse to the estimation confidence being less than or equal to thethreshold.

The estimating of the pose may include estimating the pose of the userfor the predetermined period of time corresponding to a sliding timewindow using a SLAM scheme in real time.

A non-transitory computer-readable storage medium may store instructionsthat, when executed by a processor, configure the processor to performthe method.

In another general aspect, a pose prediction apparatus includes: one ormore sensors configured to sense a motion of a user for a predeterminedperiod of time; and a processor configured to estimate a pose of theuser based on the motion of the user in real time, calculate anestimation confidence of the estimated pose, determine a weight for thepose of the user based on the estimated pose and the estimationconfidence, and predict a pose of the user after the predeterminedperiod of time, based on the weight.

The processor may be configured to: calculate a prediction error betweenthe estimated pose and the predicted pose, based on the estimationconfidence; and tune an artificial neural network (ANN) model by feedingback the prediction error, the ANN model being configured to predict apose of the user.

For the calculating of the prediction error, the processor may beconfigured to calculate the prediction error based on a comparisonresult between the estimation confidence and a predetermined threshold.

For the calculating of the prediction error, the processor may beconfigured to: calculate the prediction error in response to theestimation confidence being greater than the threshold; and adjust theprediction error based on a comparison result between the estimated poseand a pose that is estimated for a time adjacent to the predeterminedperiod of time for which the pose is estimated, in response to theestimation confidence being less than or equal to the threshold.

For the tuning of the ANN model, the processor may be configured to tunea parameter of the ANN model by numerically calculating a gradient ofthe prediction error.

For the tuning of the ANN model, the processor may be configured to tunea parameter of the ANN model using a least square scheme by groupingposes with relatively high estimation confidences as a reference set.

For the predicting of the pose, the processor may be configured to:refine the estimated pose by adjusting the weight; and predict the poseof the user after the predetermined period of time based on the refinedpose, the estimation confidence, and the prediction error that is fedback.

The apparatus may be an augmented reality (AR) glasses apparatuscomprising a display configured to visualize a virtual content objectand a real object that are matched based on the predicted pose of theuser.

In another general aspect, an augmented reality (AR) glasses apparatusincludes: one or more sensors configured to sense a motion of a user fora predetermined period of time; a processor configured to estimate apose of the user based on the motion of the user in real time,

calculate an estimation confidence of the estimated pose, determine aweight for the pose of the user based on the estimated pose and theestimation confidence, predict a pose of the user after thepredetermined period of time, based on the weight, and match a virtualcontent object and a real object based on the predicted pose of theuser; and a display configured to visualize the virtual content objectand the real object that are matched.

In another general aspect, a processor-implemented pose predictionmethod includes: determining a weight for a pose of a user based on anestimated pose of the user for a predetermined period of time and on anestimation confidence of the estimated pose; predicting, using a neuralnetwork model, a pose of the user after the predetermined period oftime, based on the weight; calculating a prediction error of thepredicted pose; and tuning the neural network model based on theprediction error.

In response to the estimation confidence being greater than a threshold,the calculating of the prediction error may include calculating theprediction error based on a comparison result between the estimated poseand the predicted pose.

In response to the estimation confidence being less than or equal to athreshold, the calculating of the prediction error may includecalculating the prediction error based on a comparison result betweenthe estimated pose and a pose that is estimated for a time adjacent tothe predetermined period of time for which the pose is estimated.

The weight may be a weight of neural network model, and the predictingmay include applying the weight to the estimated pose in response toinputting the estimated pose to the neural network model.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a simultaneous localization and mapping(SLAM) estimation error and a prediction error.

FIG. 2 illustrates an example of a concept of a pose prediction method.

FIG. 3 illustrates an example of a pose prediction method.

FIG. 4 illustrates an example of a configuration and an operation of apose prediction apparatus.

FIGS. 5 to 7 illustrate examples of calculating an estimation confidenceand refining an estimated pose.

FIGS. 8 and 9 illustrate examples of tuning an artificial neural network(ANN) model.

FIG. 10 illustrates an example of a pose prediction method.

FIG. 11 illustrates an example of a pose prediction apparatus.

Throughout the drawings and the detailed description, unless otherwisedescribed or provided, the same drawing reference numerals will beunderstood to refer to the same elements, features, and structures. Thedrawings may not be to scale, and the relative size, proportions, anddepiction of elements in the drawings may be exaggerated for clarity,illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. However, various changes,modifications, and equivalents of the methods, apparatuses, and/orsystems described herein will be apparent after an understanding of thedisclosure of this application. For example, the sequences of operationsdescribed herein are merely examples, and are not limited to those setforth herein, but may be changed as will be apparent after anunderstanding of the disclosure of this application, with the exceptionof operations necessarily occurring in a certain order. Also,descriptions of features that are known, after an understanding of thedisclosure of this application, may be omitted for increased clarity andconciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided merelyto illustrate some of the many possible ways of implementing themethods, apparatuses, and/or systems described herein that will beapparent after an understanding of the disclosure of this application.

Hereinafter, examples will be described in detail with reference to theaccompanying drawings. Various modifications may be made to theexamples. Here, the examples are not construed as limited to thedisclosure and should be understood to include all changes, equivalents,and replacements within the idea and the technical scope of thedisclosure.

The terminology used herein is for the purpose of describing examplesonly and is not to be limiting of the examples. As used herein, thesingular forms “a”, “an”, and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. As usedherein, the term “and/or” includes any one and any combination of anytwo or more of the associated listed items. It will be furtherunderstood that the terms “include,” “comprise,” and “have”, when usedherein, specify the presence of stated features, integers, steps,operations, elements, components, numbers, and/or combinations thereof,but do not preclude the presence or addition of one or more otherfeatures, integers, steps, operations, elements, components, numbers,and/or combinations thereof. The use of the term “may” herein withrespect to an example or embodiment (for example, as to what an exampleor embodiment may include or implement) means that at least one exampleor embodiment exists where such a feature is included or implemented,while all examples are not limited thereto.

Unless otherwise defined herein, all terms used herein includingtechnical or scientific terms have the same meaning as commonlyunderstood by one of ordinary skill in the art to which this disclosurepertains after and understanding of the present disclosure. It will befurther understood that terms, such as those defined in commonly-useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and thepresent disclosure, and will not be interpreted in an idealized oroverly formal sense unless expressly so defined herein.

When describing the examples with reference to the accompanyingdrawings, like reference numerals refer to like constituent elements anda repeated description related thereto will be omitted. In thedescription of examples, detailed description of well-known relatedstructures or functions will be omitted when it is deemed that suchdescription will cause ambiguous interpretation of the presentdisclosure.

Although terms of “first” or “second” are used herein to describevarious members, components, regions, layers, or sections, thesemembers, components, regions, layers, or sections are not to be limitedby these terms. Rather, these terms are only used to distinguish onemember, component, region, layer, or section from another member,component, region, layer, or section. Thus, a first member, component,region, layer, or section referred to in examples described herein mayalso be referred to as a second member, component, region, layer, orsection without departing from the teachings of the examples.

Throughout the specification, when an element, such as a layer, region,or substrate, is described as being “on,” “connected to,” or “coupledto” another element, it may be directly “on,” “connected to,” or“coupled to” the other element, or there may be one or more otherelements intervening therebetween. In contrast, when an element isdescribed as being “directly on,” “directly connected to,” or “directlycoupled to” another element, there can be no other elements interveningtherebetween. Likewise, expressions, for example, “between” and“immediately between” and “adjacent to” and “immediately adjacent to”may also be construed as described in the foregoing.

A component having a common function with a component included in oneexample is described using a like name in another example. Unlessotherwise described, description made in one example may be applicableto another example and detailed description within a duplicate range isomitted.

FIG. 1 illustrates an example of a simultaneous localization and mapping(SLAM) estimation error and a prediction error. FIG. 1 illustrates atrajectory 105 (hereinafter, referred to as a “SLAM estimated trajectory105”) of “N” poses estimated by a SLAM scheme during a sliding windowinterval, and a prediction error 109 caused by a difference between aground truth trajectory 103 and the SLAM estimated trajectory 105 in aprediction time dt 101 of the sliding window interval. The ground truthtrajectory 103 may correspond to a measured trajectory.

A pose prediction apparatus (for example, an augmented reality (AR)apparatus) may predict a head motion of a user by modeling a nonlinearmotion to reflect an actual motion of the user. For example, an ARapparatus may predict a motion through pose estimation according to aSLAM scheme which is to be used as an input for motion prediction. TheSLAM scheme may correspond to a scheme of collecting data of asurrounding environment using various sensors (for example, a radar, alidar, a global positioning system (GPS), and/or a camera) and ofcreating a map of a corresponding space based on the collected datawhile identifying a location of a device. For example, the SLAMestimated trajectory 105 may be used without a change to predict a poseof a user for the prediction time dt 101.

In this example, as shown in FIG. 1, the prediction error 109 may occurdue to the difference between the ground truth trajectory 103 and theSLAM estimated trajectory 105. The prediction error 109 may furtherincrease in response to a low confidence of an estimation result, forexample, a point 107 with a high uncertainty in the SLAM estimatedtrajectory 105.

Thus, a pose prediction method and apparatus of one or more embodimentsmay use an estimated pose and an estimation confidence of the estimatedpose together to minimize the prediction error 109 for a pose of a user,and accordingly the pose prediction method and apparatus of one or moreembodiments may predict a pose robustly against an estimation error ofthe SLAM scheme.

FIG. 2 illustrates an example of a concept of a pose prediction method.FIG. 2 illustrates a configuration of a pose prediction apparatus 200.The pose prediction apparatus 200 may include an estimation module 210and a prediction module 230.

The estimation module 210 may estimate a pose of a user in real timebased on a SLAM scheme for a predetermined period of time (for example,time @t=t0˜t1). The pose of the user estimated by the SLAM scheme mayhave six degrees of freedom (6DoF).

Also, the estimation module 210 may calculate an estimation confidenceof the estimated pose. The estimation confidence may refer to a degreeto which an estimated pose is reliable, or a confidence probability foran estimated pose. The estimation confidence may be expressed as, forexample, a value greater than or equal to “0” and less than or equal to“1”, however, there is no limitation thereto. The estimation confidencemay be represented as, for example, an uncertainty value or a confidenceprobability.

The estimation module 210 may provide the estimated pose together withthe estimation confidence (for example, an uncertainty value) of theestimated pose to the prediction module 230.

The prediction module 230 may include an artificial neural network (ANN)model that is configured to predict a pose of a user. The ANN model maycorrespond to an ANN that is pre-trained to predict a pose of a userbased on an estimated pose and an estimation confidence. The ANN modelmay include, for example, a prediction model such as a prediction model435 of FIG. 4, a non-limiting example of which will described later ingreater detail. Training an ANN may include determining and updatingweights, biases, and/or parameters between layers included in an ANN,and/or determining and updating weights, biases, and/or parametersbetween a plurality of nodes belonging to different layers amongneighboring layers. The ANN model may be configured with or include, forexample, a recurrent neural network (RNN), a long short-term memory(LSTM), and/or a regression model.

The prediction module 230 may determine a weight for a pose of a userbased on the estimated pose and the estimation confidence that arereceived from the estimation module 210. The weight for the pose of theuser may be a weight applied to a pose of a user input to the ANN modelincluded in the prediction module 230. The pose of the user may include,for example, a head motion of the user, but is not limited thereto. Inan example, to take into consideration nonlinearity of the head motion,the prediction module 230 may be configured using a deep learningstructure (for example, the LSTM and/or the RNN).

The prediction module 230 may predict a pose (e.g., predicted pose) ofthe user after a predetermined period of time (for example, time@t=t1+dt) based on the weight (e.g., based on the weight for the pose ofthe user determined based on the estimated pose and the estimationconfidence received from the estimation module 210). The pose of theuser for the time @t=t1+dt predicted by the prediction module 230 may befed back, and may be used to calculate a prediction error 250 betweenthe pose estimated by the estimation module 210 for the time @t=t0˜t1,and the pose for @t=t1+dt predicted by the prediction module 230.

The prediction module 230 of one or more embodiments may predict a poseof a user robustly against an estimation error of the SLAM scheme bytuning the ANN model according to a motion pattern of a user to bespecialized for the user. The prediction error 250 may be used to tunethe ANN model or parameters of the ANN model used by the predictionmodule 230 to predict a pose of a user.

The prediction module 230 may predict a pose robustly against anestimation error by adjusting a weight for the ANN model based on theestimation confidence of the pose estimated in the estimation module210. For example, when the estimation confidence is high, the predictionmodule 230 may tune the ANN model according to a user's behavior patternby feeding back a prediction error compared to the pose estimated basedon the SLAM scheme.

FIG. 3 illustrates an example of a pose prediction method. FIG. 3illustrates a process by which a pose prediction apparatus predicts apose of a user after a predetermined period of time through operations310 to 340.

In operation 310, the pose prediction apparatus may estimate a pose of auser for a predetermined period of time in real time. For example, thepose prediction apparatus may estimate a pose of a user for apredetermined period of time corresponding to a sliding time windowusing a SLAM scheme in real time. The predetermined period of time maycorrespond to, for example, a prediction time of the above-describedsliding time window. The pose of the user may be understood to include,for example, a location and an orientation of a user. For example, thelocation may be represented by x, y, and z coordinates, and theorientation may be represented by pitch yaw and roll angles.

For example, in operation 310, the pose prediction apparatus mayestimate a pose of a user for each time unit during 100 milliseconds(ms). When a time unit is 20 ms, the pose prediction apparatus mayestimate a pose P₁ of a user for 20 ms, a pose P₂ of the user for 40 ms,a pose P₃ of the user for 60 ms, a pose P₄ of the user for 80 ms, and apose P₅ of the user for 100 ms.

In operation 320, the pose prediction apparatus may calculate anestimation confidence (for example, an uncertainty or a confidenceprobability) of the pose estimated in operation 310. In an example, thepose prediction apparatus may calculate the uncertainty of the poseestimated in operation 310, based on an estimation covariance accordingto a SLAM scheme with a Kalman filter. In another example, the poseprediction apparatus may calculate the uncertainty of the pose estimatedin operation 310, based on a residual value between the pose estimatedin operation 310 and a value obtained by approximating the poseestimated in operation 310 using a curve fitting function.

In operation 330, the pose prediction apparatus may determine a weightfor the pose of the user based on the estimated pose and the estimationconfidence. In an example, when the estimation confidence calculated inoperation 320 is greater than a predetermined threshold, the poseprediction apparatus may set the weight to be greater than a referencevalue. In another example, when the estimation confidence is less thanor equal to the threshold, the pose prediction apparatus may adjust theweight based on a pose estimated for a time adjacent to thepredetermined period of time for which the pose is estimated inoperation 310. In operation 330, the pose prediction apparatus mayrefine the estimated pose by adjusting the weight.

Non-limiting examples in which the pose prediction apparatus calculatesan uncertainty, determines a weight for a pose of a user, and refinesthe pose will be further described below with reference to FIGS. 5 to 7.

In operation 340, the pose prediction apparatus may predict a pose ofthe user after the predetermined period of time, based on the weightdetermined in operation 330.

Depending on examples, the pose prediction apparatus may calculate aprediction error between the pose estimated in operation 310 and thepose predicted in operation 340 based on the estimation confidencecalculated in operation 320.

The pose prediction apparatus may calculate a prediction error based ona comparison result between the estimation confidence calculated inoperation 320 and a predetermined threshold. In an example, when theestimation confidence is greater than the threshold, the pose predictionapparatus may calculate a prediction error. When an uncertainty of poseestimation by the SLAM scheme obtained after reaching an actualprediction time is less than a predetermined threshold (for example,U_(required)), the pose prediction apparatus may determine that anestimate is reliable, which will be further described below. In thisexample, the pose prediction apparatus may calculate a prediction errorby comparing the pose estimated in operation 310 and the pose predictedin operation 340. The pose prediction apparatus may provide theprediction error as feedback to enable tuning of an ANN model accordingto a motion pattern of a user.

In an example, the pose prediction apparatus may tune a parameter of theANN model by numerically calculating a gradient of the prediction error.In another example, the pose prediction apparatus may tune a parameterof the ANN model using a least square scheme by grouping poses withrelatively high estimation confidences as a reference set. Examples inwhich the pose prediction apparatus tunes an ANN model will be furtherdescribed below with reference to FIGS. 8 and 9.

In another example, when the estimation confidence is less than or equalto the threshold, the pose prediction apparatus may adjust theprediction error based on a comparison result between the estimated poseand a pose that is estimated for a time adjacent to the predeterminedperiod of time for which the pose is estimated. For example, when theuncertainty of the pose estimation is greater than a predeterminedthreshold (for example, U_(max)), the pose prediction apparatus maydetermine that the estimated pose is unreliable, thereby removing theestimated pose from the sliding time window, or adjusting the weight bycomparing the estimated pose and another adjacent pose.

Through the above-described process, the pose prediction apparatus maypredict the pose of the user after the predetermined period of time,based on the refined pose, the estimation confidence, and the predictionerror that is fed back. A non-limiting example of a configuration and anoperation of the pose prediction apparatus will be further describedbelow with reference to FIG. 4.

FIG. 4 illustrates an example of a configuration and an operation of apose prediction apparatus. FIG. 4 illustrates a configuration of a poseprediction apparatus 400.

The pose prediction apparatus 400 may predict a pose of a user after apredetermined prediction time dt, that is, a time (t₁+dt), based on “N”last poses estimated for a sliding time window from t₀ to t₁ and anuncertainty of each of the estimated poses.

The pose prediction apparatus 400 may include an estimation/calculationmodule 410 and a prediction module 430. In a non-limiting example, thepose prediction apparatus 400 may include the prediction module 430 andmay not include the estimation/calculation module 410. In this example,a separate or external device may include the estimation/calculationmodule 410 that calculates an estimated pose and an estimationconfidence of the estimated pose and the estimated pose and theestimation confidence may be provided to the pose prediction apparatus400 by the separate or external device, such that the pose predictionapparatus 400 obtains the estimated pose and the estimation confidencefrom the external device. Although an operation in an example in whichthe pose prediction apparatus 400 includes both theestimation/calculation module 410 and the prediction module 430 will bemainly described below for convenience of description, othernon-limiting examples may exist in which the pose prediction apparatus400 includes the prediction module 430 and does not include theestimation/calculation module 410.

The estimation/calculation module 410 may estimate a pose of a userusing a SLAM scheme, may calculate an estimation confidence (forexample, an uncertainty) of the estimated pose, and may provide theestimated pose and the estimation confidence to the prediction module430. The estimation/calculation module 410 may include a pose estimator413 and a confidence determiner/refiner 416.

The pose estimator 413 may estimate a pose of a user in real time usingthe SLAM scheme. The confidence determiner/refiner 416 may determine aconfidence of a pose estimated within the sliding time window from t₀ tot₁ by the pose estimator 413, and may refine the estimated pose based ona weight determined for each pose. For example, the weight may be set tobe inversely proportional to the uncertainty. In an example, when theconfidence of the estimated pose is greater than a predeterminedthreshold and when an uncertainty is low, the confidencedeterminer/refiner 416 may set a weight for a corresponding pose to behigh. When the uncertainty is greater than the predetermined threshold,the confidence determiner/refiner 416 may remove the corresponding posefrom the sliding time window or may adjust a weight value for thecorresponding pose based on an adjacent estimated pose, because thecorresponding pose is unreliable. The confidence determiner/refiner 416may refine the estimated pose by the adjusted weight value.

When the estimated pose and the estimation confidence are input, theprediction module 430 may output a predicted pose after the predictiontime dt. The predicted pose may be, for example, a pose corresponding toa head motion of a user.

The prediction module 430 may adjust a weight corresponding to theestimated pose based on the estimation confidence. For example, when theestimation confidence is greater than a predetermined threshold, theprediction module 430 may tune an ANN model (for example, the predictionmodel 435) based on a behavior pattern of a user by feeding back aprediction error 440. The tuning of the ANN model may be construed asapplying the prediction model 435 to a corresponding user, orpersonalizing the prediction model 435.

The prediction module 430 may include a tuner 431, and a pose predictor433. The pose predictor 433 may include the prediction model 435. Asdescribed above, the prediction model 435 may be configured as, forexample, a regression neural network to which sequential inputs areapplied over time, or a regression model.

The tuner 431 may tune the prediction model 435, for example, parametersof the prediction model 435, based on the prediction error 440. Thetuner 431 may tune adjustable parameters by numerical calculating agradient of the prediction error 440. The parameters may include, forexample, a prediction time, and a regression parameter, but are notlimited thereto.

The tuner 431 may operate when a confidence of an estimated pose isgreater than a predetermined threshold. The tuner 431 may calculate theprediction error 440 by comparing the pose predicted by the posepredictor 433 and the pose estimated by pose estimator 413 using theSLAM scheme. The tuner 431 may provide the prediction error 440 asfeedback to the pose predictor 433 and may minimize the prediction error440. The tuner 431 may utilize an estimated pose with a relatively highconfidence through a feedback structure, to minimize an estimationerror. In an example, by feeding back an estimated pose with arelatively high confidence, the pose predictor 433 may be trained basedon a behavior pattern of a user, to perform pose prediction suitable foreach user. The tuner 431 may perform tuning based on the predictionerror 440.

The pose predictor 433 may predict a pose after the prediction time dtbased on the parameters tuned by the tuner 431, the estimationconfidence, and the prediction error 440 that is fed back.

FIG. 5 illustrates an example of calculating an estimation confidenceand refining an estimated pose. A graph 510 of FIG. 5 illustrates anexample of using a Kalman filter estimation covariance to determine apose estimation confidence.

To generate an optimal state estimate {circumflex over (x)}_(k) that isunbiased with respect to a state of a system, a Kalman filter maycombine information of two sources, for example, a predicted stateestimate x_(k) and noisy measurement values y_(k). The term “optimal”used herein may be construed to refer to minimizing a variance ofestimated states.

The graph 510 may show a process in which a measured output has aninfluence on a linear single state system identical to a state (forexample, a location of a vehicle), and measurement noise. For anuncertain measurement, the Kalman filter may calculate a state estimatethat is unbiased with a minimum variance, and an operating principle ofthe Kalman filter may be understood through a probability densityfunction shown in the graph 510.

For example, when a SLAM scheme with the Kalman filter is utilized, astate variable including a pose to be estimated, an estimationcovariance according to the state variable, and an uncertainty may berepresented as shown in Equation 1 below, for example.

x _(k)=[pvab_(a) b _(g)]T

P _(k) =E[(x _(k) −{circumflex over (x)} _(k))(x _(k) −{circumflex over(x)} _(k))^(T)]

U _(k)=sqrt(diagonal(P _(k))  Equation 1:

In Equation 1, x_(k) denotes a state variable to be estimated in a timestep k. Also, p, v, and a denote position, velocity and attitude states,respectively, and b_(a), and b_(g) denote inertial measurement unit(IMU) biases (3-dimension each). P_(k) denotes an estimation covariance,and U_(k) denotes an uncertainty in the time step k.

A pose prediction apparatus may determine a weight for a pose of a useraccording to an estimation covariance, and may refine an estimated poseby adjusting the weight.

For example, the pose prediction apparatus may calculate a weight bycalculating a difference σ_(diff)(t) between an estimation covarianceσ(t) of a time step t and a mean covariance σ_(mean) in a sliding timewindow, as shown in Equation 2 below, for example.

σ_(diff)(t)=σ(t)−σ_(mean)  Equation 2:

For example, if the difference σ_(diff)(t) is greater than apredetermined threshold, the pose prediction apparatus may set theweight to “0”. If the weight is set to “0”, the pose predictionapparatus may generate a smooth pose by interpolating a predicted poseinto an adjacent pose.

As shown in a graph 530, the pose prediction apparatus may set theweight to decrease as the difference σ_(diff)(t) increases, and may setthe weight to increase as the difference σ_(diff)(t) decreases.

FIG. 6 illustrates another example of calculating an estimationconfidence and refining an estimated pose. In FIG. 6, a graph 610 showsa pose estimated by a SLAM scheme, indicated by “SLAM output”, and avalue obtained by approximating an estimated pose by a curve fittingfunction, indicated by “Curve fitting”, and a graph 620 shows a residualvalue between a pose estimated by the SLAM scheme and a value obtainedby approximating the estimated pose by the curve fitting function.

A pose prediction apparatus may approximate an estimated pose using apredefined curve fitting function, may calculate a residual value, andmay refine a predicted pose according to a weight.

For example, the pose prediction apparatus may calculate an uncertaintyof the estimated pose, based on the residual value between the poseestimated by the SLAM scheme and a value obtained by approximating theestimated pose using the curve fitting function. A residual value usingcurve fitting may be calculated as shown in Equation 3 below, forexample.

res _(k) ={circumflex over (x)} _(k) −g _(k) , U _(k) =fn(res_(k))  Equation 3:

In Equation 3, {circumflex over (x)}_(k) denotes a value of a poseestimated by the SLAM scheme in a time step k, and g_(k) denotes a curvefitting function that approximates a head motion in the time step k.Also, res_(k) denotes a residual value calculated by an estimate in atime step and a curve fitting function.

The pose prediction apparatus may calculate an uncertainty in proportionto the residual value.

FIG. 7 illustrates another example of calculating an estimationconfidence and refining an estimated pose. FIG. 7 illustrates statevariables corresponding to an estimated pose and measurement values of asensor for a pose.

A pose prediction apparatus may refine the estimated pose by updatingstate variables corresponding to a pose at a previous point in time inaddition to a pose at a current point in time, using measurement valuesof the sensor.

The pose prediction apparatus may use a Kalman filter backpropagation ora pose graph optimization scheme to refine the estimated pose.

When a new measurement value Z_(t) is received from at least one sensor,the pose prediction apparatus may optimize and refine previouslyestimated poses (for example, X_(t-2), and X_(t-1)). The pose predictionapparatus may perform pose prediction using a refined pose estimate.

The pose prediction apparatus may refine the estimated pose by updatingposes at previous points in time together through, for example, Kalmanfilter backpropagation or pose graph optimization. The pose predictionapparatus may use a pose estimate with a higher confidence when only alatest pose estimate is used when obtaining a pose estimate within asliding time window for pose prediction by updating poses at previouspoints in time together and refining the estimated pose.

The pose prediction apparatus may adjust a weight based on adetermination of a confidence for a refined pose. For example, a Kalmanfilter covariance as described above may be used to determine aconfidence of a refined pose.

Depending on examples, the pose prediction apparatus may also performpose prediction using an estimate of a pose refined through Kalmanfilter backpropagation or pose graph optimization in a prediction modelwithout a change.

FIG. 8 illustrates an example of tuning an ANN model. A graph of FIG. 8shows a relationship between a prediction error and a control parameterof an ANN model. The ANN model may be, for example, the above-describedprediction model 435 of FIG. 4.

For example, when a confidence of a pose estimated by a SLAM scheme ishigh, a pose prediction apparatus may tune a prediction model accordingto a behavior pattern of a user by feeding back a prediction error. Thepose prediction apparatus may tune a parameter by numericallycalculating a gradient of a prediction error for an adjustable controlparameter. The adjustable control parameter may include, for example, aprediction time or a regression parameter, but is not limited thereto.

For example, the pose prediction apparatus may tune a parameter as shownin Equation 4 below.

$\begin{matrix}{p_{new} = {p_{old} - {\alpha\;\frac{\partial e}{\partial p}}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

In Equation 4, p_(old) denotes a control parameter that is not tuned,and p_(new) denotes a control parameter that is tuned. e denotes aprediction error. α denotes a parameter tuning coefficient and may havea value between “0” and “1”.

$\frac{\partial e}{\partial p}$

denotes a prediction error gradient for a control parameter.

The pose prediction apparatus may internally calculate the predictionmodel changed by ∂p for numerical calculation of the gradient. When anuncertainty of a pose estimated by a SLAM scheme is less than apredetermined threshold, the pose prediction apparatus may calculate theprediction error gradient and tune the prediction model, because anestimate is reliable.

FIG. 9 illustrates another example of tuning an ANN model. FIG. 9illustrates a process by which a pose prediction apparatus tunes an ANNmodel through operations 910 to operation 960.

In operation 910, the pose prediction apparatus may receive a poseestimate by a SLAM scheme.

In operation 920, the pose prediction apparatus may determine anestimation confidence of the pose estimate received in operation 910,and may refine an estimated pose.

In operation 930, the pose prediction apparatus may set a reference setbased on the estimation confidence determined in operation 920.

In an example, when an average pose prediction error in the referenceset is greater than a predetermined threshold, the pose predictionapparatus may tune a parameter. For example, the pose predictionapparatus may group poses with high estimation confidences as areference set, and may tune parameters of the ANN model using a leastsquare scheme. The least square scheme may be one of schemes of findinga parameter through optimization, and may correspond to a scheme offinding solutions such that a sum of squares of errors betweenapproximate solutions and actual solutions is minimized. In thisexample, the parameter may be a coefficient of an equation representinga predetermined system, or a parameter of a filter or a simplerelational expression. In an example, a parameter may correspond to acontrol parameter of the above-described ANN model, for example, aprediction model.

In operation 940, the pose prediction apparatus may perform motionprediction by the prediction model.

In operation 950, the pose prediction apparatus may calculate a motionprediction error.

In operation 960, the pose prediction apparatus may tune the predictionmodel based on the motion prediction error calculated in operation 950.If a pose prediction error is large, the pose prediction apparatus maytune the prediction model.

FIG. 10 illustrates another example of a pose prediction method. FIG. 10illustrates a process by which a pose prediction apparatus predicts apose of a user through operations 1010 to operation 1060.

In operation 1010, the pose prediction apparatus may obtain sensor databy sensing a pose of a user based on a motion of the user. The sensordata may be obtained by, for example, a camera sensor, an IMU sensor, ora gyro sensor, however, there is no limitation thereto.

In operation 1020, the pose prediction apparatus may estimate the poseof the user using a SLAM scheme based on the sensor data obtained inoperation 1010.

In operation 1030, the pose prediction apparatus may determine anestimation confidence of the pose estimated in operation 1020, and mayrefine the estimated pose. The pose prediction apparatus may set aweight of each pose based on the estimation confidence (for example, anuncertainty) of the estimated pose. The pose prediction apparatus mayrefine the estimated pose through the set weight to be robust againstthe uncertainty. Subsequently, the pose prediction apparatus may performpose prediction based on a difference between the refined pose and apredicted pose.

In operation 1040, the pose prediction apparatus may predict a poseusing a prediction model.

In operation 1050, the pose prediction apparatus may tune the predictionmodel based on a prediction error between the pose estimated inoperation 1020 and the pose predicted in operation 1040. The poseprediction apparatus may predict the pose again in operation 1040 usingthe prediction model tuned in operation 1050.

In operation 1060, the pose prediction apparatus may output the posepredicted in operation 1040.

FIG. 11 illustrates an example of a pose prediction apparatus. Referringto FIG. 11, a pose prediction apparatus 1100 may include at least onesensor 1110, a processor 1130, a memory 1150, a communication interface1170, and a display 1190. The at least one sensor 1110, the processor1130, the memory 1150, the communication interface 1170, and the display1190 may be connected to each other via a communication bus 1105.

The at least one sensor 1110 may sense a motion of a user for apredetermined period of time.

The processor 1130 may estimate a pose of the user based on the motionof the user sensed by the at least one sensor 1110 in real time. Theprocessor 1130 may calculate an estimation confidence of the estimatedpose. The processor 1130 may determine a weight for the pose of the userbased on the estimated pose and the estimation confidence. The processor1130 may predict a pose of the user after a predetermined period oftime, based on the weight.

The memory 1150 may store the motion of the user sensed by the at leastone sensor 1110. Also, the memory 1150 may store the pose of the userestimated in real time by the processor 1130 and/or the estimationconfidence of the estimated pose calculated by the processor 1130. Thememory 1150 may store the weight determined by the processor 1130 and/orthe pose of the user after the predetermined period of time predicted bythe processor 1130.

The communication interface 1170 may receive the motion of the usersensed by the at least one sensor 1110. The communication interface 1170may transmit the pose of the user after the predetermined period of timepredicted by the processor 1130 to the outside of the pose predictionapparatus 1100.

The pose prediction apparatus 1100 may selectively include the display1190. For example, when the display 1190 is included in the poseprediction apparatus 1100, the pose prediction apparatus 1100 may matcha virtual content object and a real object based on the pose of the userafter the predetermined period of time predicted by the processor 1130,and may display a matching result on the display 1190.

The pose prediction apparatus 1100 may correspond to apparatuses invarious fields, for example, an advanced driver-assistance system(ADAS), a head-up display (HUD), a three-dimensional (3D) digitalinformation display (DID), a navigation device, a neuromorphic device, a3D mobile device, a smartphone, a smart television (TV), a smartvehicle, an Internet of Things (IoT) device, a medical device, and ameasuring device. The 3D mobile device may be understood to include, forexample, a display device configured to display AR, virtual reality(VR), and/or mixed reality (MR), a head-mounted display (HMD), aface-mounted display (FMD), and AR glasses.

For example, the pose prediction apparatus 1100 is AR glasses, theprocessor 1130 may match a virtual content object and a real objectbased on a pose of a user predicted through the above-described process.In this example, the display 1190 may visualize the virtual contentobject and the real object matched by the processor 1130. The display1190 may include, for example, a flexible display, but is not limitedthereto.

Also, the processor 1130 may perform at least one method described withreference to FIGS. 1 through 10, or a scheme corresponding to the atleast one method. The processor 1130 may be a hardware-implemented poseestimation apparatus having a circuit that is physically structured toexecute desired operations. For example, the desired operations mayinclude code or instructions included in a program. Thehardware-implemented pose estimation apparatus may include, for example,a microprocessor, a central processing unit (CPU), a graphics processingunit (GPU), a processor core, a multi-core processor, a multiprocessor,an application-specific integrated circuit (ASIC), a field-programmablegate array (FPGA), and a neural processing unit (NPU).

The processor 1130 may execute a program and may control the poseprediction apparatus 1100. Code of the program executed by the processor1130 may be stored in the memory 1150.

As described above, the memory 1150 may store a variety of informationgenerated in a processing process of the above-described processor 1130.Also, the memory 1150 may store a variety of data and programs. Thememory 1150 may include, for example, a volatile memory or anon-volatile memory. The memory 1150 may include a high-capacity storagemedium such as a hard disk to store a variety of data.

The pose prediction apparatuses, estimation modules, prediction modules,estimation/calculation modules, pose estimators, confidencedeterminer/refiners, tuners, pose predictors, sensors, processors,memories, communication interfaces, displays, pose prediction apparatus200, estimation module 210, prediction module 230, pose predictionapparatus 400, estimation/calculation module 410, pose estimator 413,confidence determiner/refiner 416, prediction module 430, tuner 431,pose predictor 433, prediction model 435, pose prediction apparatus1100, at least one sensor 1110, processor 1130, memory 1150,communication interface 1170, display 1190, and other apparatuses,units, modules, devices, and components described herein with respect toFIGS. 1-11 are implemented by or representative of hardware components.Examples of hardware components that may be used to perform theoperations described in this application where appropriate includecontrollers, sensors, generators, drivers, memories, comparators,arithmetic logic units, adders, subtractors, multipliers, dividers,integrators, and any other electronic components configured to performthe operations described in this application. In other examples, one ormore of the hardware components that perform the operations described inthis application are implemented by computing hardware, for example, byone or more processors or computers. A processor or computer may beimplemented by one or more processing elements, such as an array oflogic gates, a controller and an arithmetic logic unit, a digital signalprocessor, a microcomputer, a programmable logic controller, afield-programmable gate array, a programmable logic array, amicroprocessor, or any other device or combination of devices that isconfigured to respond to and execute instructions in a defined manner toachieve a desired result. In one example, a processor or computerincludes, or is connected to, one or more memories storing instructionsor software that are executed by the processor or computer. Hardwarecomponents implemented by a processor or computer may executeinstructions or software, such as an operating system (OS) and one ormore software applications that run on the OS, to perform the operationsdescribed in this application. The hardware components may also access,manipulate, process, create, and store data in response to execution ofthe instructions or software. For simplicity, the singular term“processor” or “computer” may be used in the description of the examplesdescribed in this application, but in other examples multiple processorsor computers may be used, or a processor or computer may includemultiple processing elements, or multiple types of processing elements,or both. For example, a single hardware component or two or morehardware components may be implemented by a single processor, or two ormore processors, or a processor and a controller. One or more hardwarecomponents may be implemented by one or more processors, or a processorand a controller, and one or more other hardware components may beimplemented by one or more other processors, or another processor andanother controller. One or more processors, or a processor and acontroller, may implement a single hardware component, or two or morehardware components. A hardware component may have any one or more ofdifferent processing configurations, examples of which include a singleprocessor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated in FIGS. 1-11 that perform the operationsdescribed in this application are performed by computing hardware, forexample, by one or more processors or computers, implemented asdescribed above executing instructions or software to perform theoperations described in this application that are performed by themethods. For example, a single operation or two or more operations maybe performed by a single processor, or two or more processors, or aprocessor and a controller. One or more operations may be performed byone or more processors, or a processor and a controller, and one or moreother operations may be performed by one or more other processors, oranother processor and another controller. One or more processors, or aprocessor and a controller, may perform a single operation, or two ormore operations.

Instructions or software to control computing hardware, for example, oneor more processors or computers, to implement the hardware componentsand perform the methods as described above may be written as computerprograms, code segments, instructions or any combination thereof, forindividually or collectively instructing or configuring the one or moreprocessors or computers to operate as a machine or special-purposecomputer to perform the operations that are performed by the hardwarecomponents and the methods as described above. In one example, theinstructions or software include machine code that is directly executedby the one or more processors or computers, such as machine codeproduced by a compiler. In another example, the instructions or softwareincludes higher-level code that is executed by the one or moreprocessors or computer using an interpreter. The instructions orsoftware may be written using any programming language based on theblock diagrams and the flow charts illustrated in the drawings and thecorresponding descriptions used herein, which disclose algorithms forperforming the operations that are performed by the hardware componentsand the methods as described above.

The instructions or software to control computing hardware, for example,one or more processors or computers, to implement the hardwarecomponents and perform the methods as described above, and anyassociated data, data files, and data structures, may be recorded,stored, or fixed in or on one or more non-transitory computer-readablestorage media. Examples of a non-transitory computer-readable storagemedium include read-only memory (ROM), random-access programmable readonly memory (PROM), electrically erasable programmable read-only memory(EEPROM), random-access memory (RAM), dynamic random access memory(DRAM), static random access memory (SRAM), flash memory, non-volatilememory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs,DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-rayor optical disk storage, hard disk drive (HDD), solid state drive (SSD),flash memory, a card type memory such as multimedia card micro or a card(for example, secure digital (SD) or extreme digital (XD)), magnetictapes, floppy disks, magneto-optical data storage devices, optical datastorage devices, hard disks, solid-state disks, and any other devicethat is configured to store the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and provide the instructions or software and any associated data,data files, and data structures to one or more processors or computersso that the one or more processors or computers can execute theinstructions. In one example, the instructions or software and anyassociated data, data files, and data structures are distributed overnetwork-coupled computer systems so that the instructions and softwareand any associated data, data files, and data structures are stored,accessed, and executed in a distributed fashion by the one or moreprocessors or computers.

While this disclosure includes specific examples, it will be apparentafter an understanding of the disclosure of this application thatvarious changes in form and details may be made in these exampleswithout departing from the spirit and scope of the claims and theirequivalents. The examples described herein are to be considered in adescriptive sense only, and not for purposes of limitation. Descriptionsof features or aspects in each example are to be considered as beingapplicable to similar features or aspects in other examples. Suitableresults may be achieved if the described techniques are performed in adifferent order, and/or if components in a described system,architecture, device, or circuit are combined in a different manner,and/or replaced or supplemented by other components or theirequivalents. Therefore, the scope of the disclosure is defined not bythe detailed description, but by the claims and their equivalents, andall variations within the scope of the claims and their equivalents areto be construed as being included in the disclosure.

What is claimed is:
 1. A processor-implemented pose prediction methodcomprising: estimating a pose of a user for a predetermined period oftime in real time; calculating an estimation confidence of the estimatedpose; determining a weight for the pose of the user based on theestimated pose and the estimation confidence; and predicting a pose ofthe user after the predetermined period of time, based on the weight. 2.The method of claim 1, further comprising: calculating a predictionerror between the estimated pose and the predicted pose, based on theestimation confidence; and tuning an artificial neural network (ANN)model by feeding back the prediction error, the ANN model beingconfigured to predict a pose of the user.
 3. The method of claim 2,wherein the calculating of the prediction error comprises calculatingthe prediction error based on a comparison result between the estimationconfidence and a predetermined threshold.
 4. The method of claim 3,wherein the calculating of the prediction error based on the comparisonresult comprises: calculating the prediction error in response to theestimation confidence being greater than the threshold; and adjustingthe prediction error based on a comparison result between the estimatedpose and a pose that is estimated for a time adjacent to thepredetermined period of time for which the pose is estimated, inresponse to the estimation confidence being less than or equal to thethreshold.
 5. The method of claim 2, wherein the tuning of the ANN modelcomprises tuning a parameter of the ANN model by numerically calculatinga gradient of the prediction error.
 6. The method of claim 2, whereinthe tuning of the ANN model comprises tuning a parameter of the ANNmodel using a least square scheme by grouping poses with relatively highestimation confidences as a reference set.
 7. The method of claim 1,wherein the calculating of the estimation confidence comprisescalculating an uncertainty of the estimated pose, based on an estimationcovariance according to a simultaneous localization and mapping (SLAM)scheme with a Kalman filter.
 8. The method of claim 1, wherein thecalculating of the estimation confidence comprises calculating anuncertainty of the estimated pose, based on a residual value between theestimated pose and a value obtained by approximating the estimated poseusing a curve fitting function.
 9. The method of claim 2, wherein thepredicting of the pose comprises: refining the estimated pose byadjusting the weight; and predicting the pose of the user after thepredetermined period of time based on the refined pose, the estimationconfidence, and the prediction error that is fed back.
 10. The method ofclaim 1, wherein the determining of the weight comprises: setting theweight to be greater than a reference value in response to theestimation confidence being greater than a predetermined threshold; andadjusting the weight based on a pose that is estimated for a timeadjacent to the predetermined period of time for which the pose isestimated, in response to the estimation confidence being less than orequal to the threshold.
 11. The method of claim 1, wherein theestimating of the pose comprises estimating the pose of the user for thepredetermined period of time corresponding to a sliding time windowusing a SLAM scheme in real time.
 12. A non-transitory computer-readablestorage medium storing instructions that, when executed by a processor,configure the processor to perform the method of claim
 1. 13. A poseprediction apparatus comprising: one or more sensors configured to sensea motion of a user for a predetermined period of time; and a processorconfigured to estimate a pose of the user based on the motion of theuser in real time, calculate an estimation confidence of the estimatedpose, determine a weight for the pose of the user based on the estimatedpose and the estimation confidence, and predict a pose of the user afterthe predetermined period of time, based on the weight.
 14. The apparatusof claim 13, wherein the processor is configured to: calculate aprediction error between the estimated pose and the predicted pose,based on the estimation confidence; and tune an artificial neuralnetwork (ANN) model by feeding back the prediction error, the ANN modelbeing configured to predict a pose of the user.
 15. The apparatus ofclaim 14, wherein, for the calculating of the prediction error, theprocessor is configured to calculate the prediction error based on acomparison result between the estimation confidence and a predeterminedthreshold.
 16. The apparatus of claim 15, wherein, for the calculatingof the prediction error, the processor is configured to: calculate theprediction error in response to the estimation confidence being greaterthan the threshold; and adjust the prediction error based on acomparison result between the estimated pose and a pose that isestimated for a time adjacent to the predetermined period of time forwhich the pose is estimated, in response to the estimation confidencebeing less than or equal to the threshold.
 17. The apparatus of claim14, wherein, for the tuning of the ANN model, the processor isconfigured to tune a parameter of the ANN model by numericallycalculating a gradient of the prediction error.
 18. The apparatus ofclaim 14, wherein, for the tuning of the ANN model, the processor isconfigured to tune a parameter of the ANN model using a least squarescheme by grouping poses with relatively high estimation confidences asa reference set.
 19. The apparatus of claim 13, wherein, for thepredicting of the pose, the processor is configured to: refine theestimated pose by adjusting the weight; and predict the pose of the userafter the predetermined period of time based on the refined pose, theestimation confidence, and the prediction error that is fed back. 20.The apparatus of claim 13, wherein the apparatus is an augmented reality(AR) glasses apparatus comprising a display configured to visualize avirtual content object and a real object that are matched based on thepredicted pose of the user.
 21. An augmented reality (AR) glassesapparatus comprising: one or more sensors configured to sense a motionof a user for a predetermined period of time; a processor configured toestimate a pose of the user based on the motion of the user in realtime, calculate an estimation confidence of the estimated pose,determine a weight for the pose of the user based on the estimated poseand the estimation confidence, predict a pose of the user after thepredetermined period of time, based on the weight, and match a virtualcontent object and a real object based on the predicted pose of theuser; and a display configured to visualize the virtual content objectand the real object that are matched.
 22. A processor-implemented poseprediction method comprising: determining a weight for a pose of a userbased on an estimated pose of the user for a predetermined period oftime and on an estimation confidence of the estimated pose; predicting,using a neural network model, a pose of the user after the predeterminedperiod of time, based on the weight; calculating a prediction error ofthe predicted pose; and tuning the neural network model based on theprediction error.
 23. The method of claim 22, wherein, in response tothe estimation confidence being greater than a threshold, thecalculating of the prediction error comprises calculating the predictionerror based on a comparison result between the estimated pose and thepredicted pose.
 24. The method of claim 22, wherein, in response to theestimation confidence being less than or equal to a threshold, thecalculating of the prediction error comprises calculating the predictionerror based on a comparison result between the estimated pose and a posethat is estimated for a time adjacent to the predetermined period oftime for which the pose is estimated.
 25. The method of claim 22,wherein the weight is a weight of neural network model, and thepredicting comprises applying the weight to the estimated pose inresponse to inputting the estimated pose to the neural network model.